@thi.ng/iterators
This project is part of the
@thi.ng/umbrella monorepo.
About
Collection of ~50 composable, iterator-centric data processing
functions, largely implemented as ES6 iterators / generators, inspired
by
clojure.core
API. Written in TypeScript.
Installation
yarn add @thi.ng/iterators
All functions are defined as sub-modules and re-exported to allow the
full library to be imported if desired. Sub-module file names use Kebab
case, whereas function names are in Camel case.
import * as ti from "@thi.ng/iterators";
import { partitionBy } from "@thi.ng/iterators/partition-by";
Dependencies
API
In alphabetical order:
butLast<T>(input: Iterable<T>) => IterableIterator<T>
Yields iterator of all but the last value of input.
[...ti.butLast([])]
[...ti.butLast([1])]
[...ti.butLast([1,2])]
[...ti.butLast([1,2,3])]
[...ti.butLast("hello")]
[...ti.butLast(ti.range(10))]
cached<T>(input: Iterable<T>) => CachedIterableIterator<T>
Consumes and lazily caches values of a finite input and returns
a no-arg function, which when called return new iterator. These iterator
instances always start from the beginning of the cache and allows for
separate states and sharing of cached results among arbitrary number of
iterators. The original input is only consumed when attempting to read
beyond current cache boundary.
c = ti.cached(ti.range(10));
a = c();
b = c();
a.next();
b.next();
b.next();
a.next();
angles = ti.cached(ti.range(0, 360, 45));
[...angles()]
ti.zip(
angles(),
ti.map(
ti.juxt(Math.sin, Math.cos),
ti.map(
(x)=> x * 180 / Math.PI,
angles()
)
)
);
concat<T>(...inputs: Iterable<T>[]) => IterableIterator<T>
Produces iterator yielding lazy concatenation of given iterables.
For practical purposes none but the last input should be infinite. Any
null
or undefined
input arguments are skipped in the output.
[... ti.concat([1, 2, 3], [10, 20, 30], [100, 200, 300])]
[...ti.concat.apply(null, ["abc", null, [1, 2, 3]])]
constantly<T>(x: T) => (...args: any[]) => T
Helper function returning a new fn which takes any number of args and
always returns x
.
iter = ti.takeWhile(x => x < 0.98, ti.repeatedly(()=> Math.random()));
ti.reduce((a, b)=> a + b, 0, ti.map(ti.constantly(1), iter))
times10 = (flt, coll) => [...ti.map(x => x * 10, ti.filter(flt, coll))];
filterModes = {
odd: x => (x % 2) == 1,
even: x => (x % 2) == 0,
all: ti.constantly(true)
};
times10(filterModes[???], [1, 2, 3]);
consume(input: Iterator<any>, n?: number) => void
Helper function. Consumes & discards items from iterator (possibly for
side effects) and optionally only up to the given number of steps.
cycle<T>(input: Iterable<T>) => IterableIterator<T>
Produces iterator which lazily caches & infinitely repeats
sequence of input. Important: Input MUST be finite, use take
to
truncate input or output if necessary.
[... ti.take(10, ti.cycle(ti.range(3)))]
dedupe<T>(input: Iterable<T>) => IterableIterator<T>
Produces iterator which discards successive duplicate values from input.
Important: Uses ===
for equality checks.
[... ti.dedupe([1, 2, 2, 3, 4, 4, 4, 3])]
dedupeWith<T>(equiv: (a:T, b: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Like dedupe
, but uses given function equiv
to determine equivalence
of successive values.
var coll = [{ a: 1 }, { a: 1, b: 2 }, { a: 2, b: 2 }, { a: 2, b: 2 }, { a: 3 }];
var eq = (a, b) => a.a === b.a;
[...ti.dedupeWith(eq, coll)]
dense<T>(input: Iterable<T>) => IterableIterator<T>
Yields iterator of all non-null
and non-undefined
values of input
(e.g. a sparse array).
var a = []
a[10] = 1;
a[20] = 2;
[...ti.dense(a)]
drop<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Consumes & discards up to n
items from input and returns remaining
(possibly exhausted) iterator.
[... ti.drop(5, ti.range(10))]
[... ti.drop(5, ti.range(3))]
[... ti.take(5, ti.drop(5, ti.range()))]
dropNth<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Produces iterator which drops every n
th item from input.
[... ti.dropNth(2, ti.range(10))]
[... ti.dropNth(3, ti.range(10))]
dropWhile<T>(pred: (x: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Consumes input and calls pred
for each item. Discards all items whilst
pred
returns true
, then returns remaining (possibly exhausted)
iterator.
[... ti.dropWhile((x) => x < 5, ti.range(10))]
every<T>(pred: (x: T) => boolean, input: Iterable<T>) => boolean
Consumes input and calls pred
for each item. When pred
returns not
true
, process stops and returns false
itself. When all items pass
the predicate, the function returns true
.
If input is empty/exhausted prior to execution, every
will return
false
.
var nums = ti.iterator([2, 4, 6, 8, 10]);
ti.every((x) => (x % 2) === 0, nums);
nums.next()
nums = ti.iterator([2, 3, 4]);
ti.every((x) => (x % 2) === 0, nums);
nums.next()
ti.every((x) => true, [])
filter<T>(pred: (x: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Consumes input and calls pred
for each item. Yields iterator of items
for which pred
returned true
.
var multOf3 = (x) => (x % 3) === 0;
[... ti.filter(multOf3, ti.range(10))];
flatten(input: Iterable<any>, flatObjects = true) => IterableIterator<any>
Produces iterator which recursively flattens input (using
flattenWith
). Important: Recursion only applies to iterable types
(excluding strings) and objects (enabled by default, using
objectIterator
, see below).
[... ti.flatten([1, [2, 3], [4, [5, ["abc"]]]])]
[...ti.flatten([{ a: 1 }, { a: 23, b: 42, c: [1, 2, 3] }])]
[...ti.flatten([{ a: 1 }, [1, 2, 3], { a: 23, b: 42, c: [1, 2, 3] }], false)]
flattenWith(tx: (x: any) => any, input: Iterable<any>) => IterableIterator<any>
Produces iterator which selectively & recursively flattens input. The
first arg tx
is a transformation fn called for each
non-null/undefined
value taken from the input. It's used to check and
possibly obtain an iteratable value for further flattening. The
transformer must return undefined
if the value can't or shouldn't be
flattened. If a value is returned it MUST be iterable.
The default transformer used by flatten
is:
let defaultTx = x =>
(typeof x !== "string" && (maybeIterator(x) || maybeObjectIterator(x))) ||
undefined;
let tx = x => typeof x == "string" ? ti.map(x => x.charCodeAt(0), x) : ti.maybeIterator(x);
[...ti.flattenWith(tx, ["ROOT", undefined, ["CHILD_1", null, ["CHILD_2"]]])]
fnil(fn: (...args: any[]) => any, ...ctors: (() => any)[]) => (...args: any[]) => any
Takes a function fn
and up to 3 ctor
functions. Produces a new
function that calls fn
, replacing a null
or undefined
arg with the
value obtained by calling its related positional ctor
fn (e.g. the
first ctor is used to supply first arg, etc.).
The function fn
can take any number of arguments, however only the
first 3 are being potentially patched, how many depends on the number of
ctor
fns supplied.
hello = (greet, name) => `${greet}, ${name}!`;
helloEN = ti.fnil(hello, () => "Hi", () => "user");
helloDE = ti.fnil(hello, () => "Hallo", () => `Teilnehmer #${(Math.random()*100)|0}`);
helloEN();
helleEN(null,"toxi");
helloEN("Howdy","toxi");
helloDE();
inc = x => x + 1
adder = ti.fnil(inc, () => 0);
adder();
adder(null);
adder(10);
updateWith = f => (obj, id) => { return obj[id] = f(obj[id]), obj; }
ti.reduce(updateWith(adder), {}, "abracadabra");
fork<T>(input: Iterable<T>, cacheLimit = 16) => () => IterableIterator<T>
Similar to cached
, this function allows multiple consumers of a single
input, however is only using a sliding window of N cached values
(cached
stores the entire input).
fork
returns a no-arg function, which returns a new forked iterator
when called. There's no limit to the number of active forks.
Important: The use case for fork
is synchronized consumption at
similar speeds (up to cacheLimit
divergence). The cache is shared by
all forks. If one of the forks consumes the input faster than the
given cacheLimit
, the other forks will lose intermediate values. If in
doubt, increase the cache limit to a higher value (default 16). The
cache uses
@thi.ng/dcons
to avoid unnecessary copying during window sliding.
src = ti.partition(5, 1, ti.repeatedly(()=> (Math.random() * 100) | 0, 10));
f = ti.fork(src, 1);
raw = f();
sma = ti.map((part)=> ti.reduce((a, b) => a + b, 0, part) / part.length, f());
min = ti.map((part)=> ti.reduce((a, b) => Math.min(a, b), 100, part), f());
max = ti.map((part)=> ti.reduce((a, b) => Math.max(a, b), -1, part), f());
for(let part of raw) {
console.log(`part: ${part}, avg: ${sma.next().value}, min: ${min.next().value}, ${max.next().value}`);
}
frequencies<T>(input: Iterable<T>, key?: (v: T) => any): IterableIterator<FrequencyPair<T>[]>
Consumes input, applies key
fn (if given) to each item and yields
iterator of 2-element tuples, each [key, freq]
(where freq
is the
number of times the item occurred). Important: The input MUST be
finite. Implementation uses JSON.stringify
to determine key equality.
If no key
fn is given, the original values will be used as key.
[... ti.frequencies([[1,2], [2,3], [1,2], [2,4]])]
[... ti.frequencies([1, 2, 3, 4, 5, 9, 3], (x) => x & ~1)]
var isLetter = (x) => /[a-z]/i.test(x);
[... ti.frequencies(ti.filter(isLetter, "hello world!"))].sort((a, b) => b[1] - a[1])
groupBy<T>(key: (v) => any, input: Iterable<T>) => { [id: string]: T[] }
Consumes input, applies key
fn to each item and returns object of
items grouped by result of key
fn. Important: The input MUST be
finite. Implementation uses JSON.stringify
to determine key equality.
groupBy((x) => x & ~1, [1, 2, 3, 4, 5, 9, 3])
groupBy((x) => x.toUpperCase(), "AbRaCadaBra")
identity<T>(x: T) => T
Helper function. Simply returns its argument.
ti.identity()
ti.identity(null)
ti.identity(42)
tests = [true, true, undefined, true]
ti.every(ti.identity, tests);
ti.every(ti.fnil(ti.identity, () => true), tests);
indexed<T>(input: Iterable<T>) => IterableIterator<[number, T]>
Yields iterator producing [index, value]
pairs of input.
[...ti.indexed("hello")]
interleave(...inputs: Iterable<any>[]) => IterableIterator<any>
Takes an arbitrary number of iterators and yields iterator of
interleaved items from each input. Terminates as soon as one of the
inputs is exhausted.
[... ti.interleave(ti.range(), ti.range(100, 200), ti.range(200, 205))]
interpose(x: any, input: Iterable<any>) => IterableIterator<any>
Produces an iterator which injects x
inbetween each item from input
iter
.
[... ti.interpose("/", ti.range(5))]
iterate<T>(fn: (x: T) => T, seed: T) => IterableIterator<T>
Produces an iterator of the infinite results of iteratively applying
fn
to seed
.
Important: Use take
to truncate sequence.
[... ti.take(10, ti.iterate((x) => x * 2, 1))]
iterator<T>(x: Iterable<T>) => Iterator<T>
Syntax sugar. Returns x[Symbol.iterator]()
. Most functions in this
module call this automatically for each input.
juxt<T>(...fns: ((x: T) => any)[]) => (x: T) => any[]
Takes arbitrary number of functions and returns new function, which
takes single argument x
and produces array of result values of
applying each input function to x
(juxtoposition of the given
transformation functions).
var kernel = ti.juxt(
(x) => x - 1,
(x) => x,
(x) => x + 1
);
kernel(1)
[... ti.map(kernel, ti.range(3))]
last<T>(input: Iterable<T>) => T
Consumes a finite iterator and returns last item.
Important: Never attempt to use with an infinite input!
ti.last(ti.range(10))
ti.last(ti.take(10, ti.range()))
map<T>(fn: (...args: any[]) => T, ...inputs: Iterable<any>[]) => IterableIterator<T>
Consumes an arbitrary number of iterators and applies fn
to each of
their values. Produces iterator of results. Iteration stops as soon as
one of the inputs is exhausted. The mapping fn
should accept as many
arguments as there are inputs to map
. Provides a fast path for single
input mapping.
[... ti.map((x)=> x*10, ti.range(10))]
[... ti.map((x, y, z) => [x, y, z], ti.range(5), ti.range(0, 100, 10), ti.range(0, 1000, 100))]
mapcat<T>(fn: (...args: any[]) => Iterable<T>, ...inputs: Iterable<any>[]) => IterableIterator<T>
Like map
, but expects mapping fn to return an iterable result and
produces iterator which yields the flat concatenation of results (only
the first level of nesting is removed). null
or undefined
values
returned by the mapping fn are skipped in the output.
[... ti.mapcat((x) => ti.repeat(x, 3), "hello")]
[...ti.mapcat((x) => x < 5 ? ti.repeat(x,x) : null, ti.range(10))]
mapIndexed<T>(fn: (i: number, ...args: any[]) => T[], ...inputs: Iterable<any>[]) => IterableIterator<T>
Like map
, but too passes a monotonically increasing index
as first
argument to mapping fn.
[... ti.mapIndexed((i, a, b) => [i, a, b], "hello", "there")]
maybeIterator(x: any) => any
Helper function, returning arg's iterator (if present) or else
undefined
.
maybeObjectIterator(x: any) => any
Helper function, checks if x
is object-like (but no generator) and
returns objectIterator(x)
or else undefined
.
objectIterator(x: any) => IterableIterator<any[]>
Produces iterator of an object"s key/value pairs.
[... ti.objectIterator({a: 23, b: 42, c: [1, 2, 3]})]
partition<T>(n: number, step: number, input: Iterable<T>, all = false) => IterableIterator<T[]>
Produces iterator of fixed size partitions/chunks of input values.
Produces overlapping partitions, if step
< partition size n
. Unless
the optional all
flag is enabled, returns only completely filled
partitions. If all = true
, the last partition produced may have less
than n
items (though never empty).
[... ti.partition(3, 3, ti.range(10))]
[... ti.partition(3, 3, ti.range(10), true)]
[... ti.partition(3, 1, ti.range(10))]
[... ti.partition(3, 5, ti.range(10))]
partitionBy<T>(fn: (x: T) => any, input: Iterable<T>) => IterableIterator<T[]>
Produces iterator of partitions/chunks of input values. Applies fn
to
each item and starts new partition each time fn
returns new result.
[... ti.partitionBy((x) => x / 5 | 0, ti.range(11))]
randomSample<T>(prob: number, input: Iterable<T>) => IterableIterator<T>
Produces iterator which consumes input and yields values with given
probability (0 .. 1 range).
[... ti.randomSample(0.1, ti.range(100))]
range(from?: number, to?: number, step?: number) => IterableIterator<number>
Produces iterator of monotonically increasing or decreasing values with
optional step
value.
- If called without arguments, produces values from 0 .. +∞.
- If called with 1 arg: 0 ... n (exclusive)
- If called with 2 arg:
from
... to
(exclusive)
If from
> to
and no step
is given, a step
of -1
is used.
[... ti.take(5, ti.range())]
[... ti.range(5)]
[... ti.range(100, 105)]
[... ti.range(5,0)]
[... ti.range(0, 50, 10)]
[... ti.range(50, -1, -10)]
reduce<A, B>(rfn: (acc: B, x: A) => B | ReducedValue<B>, acc: B, input: Iterable<A>) => B
Consumes and reduces input using reduction function rfn
, then returns
reduced result value. rfn
can abort reduction process by returning a
value wrapped using reduced(x)
. If this is done, then this value is
unwrapped and returned as final result.
If input is empty, returns initial acc
umulator arg.
ti.reduce((acc, x) => acc + x, 0, ti.range(10))
ti.reduce((acc, x) => { return acc += x, acc >= 15 ? ti.reduced(acc) : acc }, 0, ti.range())
reductions<A, B>(rfn: (acc: B, x: A) => B | ReducedValue<B>, acc: B, input: Iterable<A>) => IterableIterator<B[]>
Like reduce
, but yields an iterator of all intermediate reduction
results. Always yields at least initial acc
umulator arg, even if input
is empty.
Thus, the result is the equivalent of an exclusive scan
operation
(with the exception of possible early bail out via reduced
).
[... ti.reductions((acc, x) => acc + x, 0, ti.range(10))]
[... ti.reductions((acc, x) => { return acc += x, acc >= 15 ? ti.reduced(acc) : acc }, 0, ti.range())]
reduced<T>(x: T) => ReducedValue<T>
For use inside reduction functions only. Wraps result in marker type to
cause early termination of reduction (see example above).
repeat<T>(x: T, n?: number) => IterableIterator<T>
Produces an iterator of infinite (by default) repetitions of value x
.
If n
is given, produces only that many values.
[... ti.take(5, ti.repeat(42))]
[... ti.repeat(42, 5)]
repeatedly<T>(fn: () => T, n?: number) => IterableIterator<T>
Produces an iterator of infinite (by default) results of calling the
no-arg fn
repeatedly. If n
is given, produces only that many values.
[... ti.take(3, ti.repeatedly(() => Math.random()))]
[... ti.repeatedly(() => Math.random(), 3)]
reverse<T>(input: Iterable<T>) => IterableIterator<T>
Yields iterator lazily producing reverse result order of input
(must be finite). If input is not already array-like (strings are
for this purpose), the function first consumes & caches input as array.
[...ti.reverse([1, 2, 3])]
[...ti.reverse("hello")]
[...ti.reverse(ti.take(10, ti.iterate(x => x * 2, 1)))]
some<T>(pred: (x: T) => boolean, input: Iterable<T>) => T
Consumes iterator and calls pred
for each item. When pred
returns
true
, process stops and returns this first successful item. When none
of the items pass the predicate, the function returns undefined
.
var nums = ti.iterator([1, 2, 3]);
ti.some((x) => (x % 2) === 0, nums);
nums.next()
nums = ti.iterator([1, 2, 3]);
ti.some((x) => x > 3, nums);
nums.next()
take<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Produces iterator of the first n
values of input (or less than n
, if
input is too short...)
[... ti.take(3, ti.range())]
takeNth<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Produces an iterator only yielding every n
th item from input.
[... ti.takeNth(3, ti.range(10))]
takeWhile<T>(pred: (x: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Produces iterator which calls pred
for each item of input and
terminates as soon as pred
returns false
.
Important: Due to lack of look-ahead in the ES6 iterator API, the
value failing the given pred
will be lost when working with the
original iterator after takeWhile
.
var input = ti.range(10);
[... ti.takeWhile((x)=> x < 5, input)]
[... input]
takeLast<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Consumes input and produces iterator of the last n
values of input (or
less than n
, if input is too short...)
Important: Never attempt to use with infinite inputs!
[... ti.takeLast(5, ti.range(1000))]
[... ti.takeLast(5, ti.range(3))]
walk(fn: (x: any) => void, input: Iterable<any>, postOrder = false) => void
walk(fn: (x: any) => void, children: (x: any) => any, input: Iterable<any>, postOrder = false) => void
Recursively walks input and applies fn
to each element (for in-place
editing or side effects). Only iterable values and objects (but not
strings) are traversed further. Traversal is pre-order by default, but
can be changed to post-order via last arg.
Note: Object traversal is done via objectIterator()
and so will
include pairs of [k, v]
values.
The optional children
fn can be used to select child values of the
currently visited value. If this function is given then only its return
values will be traversed further (with the same constraint as mentioned
above). If the fn returns null
or undefined
, no children will be
visited.
let doc = {
tag: "svg",
content: [
{
tag: "g",
content: [
{ tag: "rect" },
{ tag: "circle" }
]
},
{ tag: "circle", attr: { fill: "red" } }
]
};
let circleTX = x => {
if (x.tag === "circle") {
x.attr = x.attr || {};
x.attr.x = Math.random()*100;
x.attr.y = Math.random()*100;
x.attr.r = 5;
}
};
ti.walk(circleTX, (x) => x.content, doc);
doc.content[0].content[1]
doc.content[1]
walkIterator(input: Iterable<any>, postOrder = false) => IterableIterator<any>
walkIterator(input: Iterable<any>, children: (x: any) => any, postOrder = false) => IterableIterator<any>
Yields an iterator performing either pre-order (default) or post-order
traversal of
input. Only iterable values and objects (but not strings) are traversed
further.
Note: Object traversal is done via objectIterator()
and so will
include pairs of [k, v]
values.
The optional children
fn can be used to select child values of the
currently visited value. If this function is given then only its return
values will be traversed further (with the same constraint as mentioned
above). If the fn returns null
or undefined
, no children will be
visited.
[...ti.map(JSON.stringify, ti.walkIterator([[[1, [2]], [3, [4]]], [5]], false))]
[...ti.map(JSON.stringify, ti.walkIterator([[[1, [2]], [3, [4]]], [5]], true))]
zip(keys: Iterable<any>, vals: Iterable<any>, target?: any) => any
Takes an iterator of keys and iterator of values, pairwise combines
items from each input and associates them as key-value mappings in a
given target object (if target
is missing, returns new object). Stops
as soon as either input is exhausted.
ti.zip("abcdef", ti.range())
ti.zip(ti.range(5,10), ti.range(100,200), new Uint8Array(16))
var langs=[
{id: "js", name: "JavaScript"},
{id: "clj", name: "Clojure"},
{id: "ts", name: "TypeScript"}
];
ti.zip(ti.map((x)=> x.id, langs), langs)
Authors
License
© 2017-2018 Karsten Schmidt // Apache Software License 2.0